Automatic Facial Action Unit Recognition by Modeling Their Semantic And Dynamic Relationships

A system that could automatically analyze the facial actions in real-time has applications in a wide range of different fields. The previous facial action unit (AU) recognition approaches often recognize AUs or certain AU combinations individually and statically, ignoring the semantic relationships among AUs and the dynamics of AUs. Hence, these approaches cannot always recognize AUs reliably, robustly, and consistently due to the richness, ambiguity, and the dynamic nature of facial actions. In this work, a novel AU recognition system is proposed to systematically account for the relationships among AUs and their temporal evolutions based on a dynamic Bayesian network (DBN). The DBN provides a coherent and unified hierarchical probabilistic framework to represent probabilistic relationships among various AUs and to account for the temporal changes in facial action development. Within the proposed system, robust computer vision techniques are used to obtain AU measurements. And such AU measurements are then applied as evidence to the DBN for inferring various AUs. The experiments show that the integration of AU relationships and AU dynamics with AU measurements yields significant improvement of AU recognition, especially under realistic environments including illumination variation, face pose variation, and occlusion. Y. Tong General Electric Global Research Center, Schenectady, NY e-mail: [email protected] W. Liao The Thomson Corporation, Eagan, MN e-mail: wenhui.liao@thomson Q. Ji ( ) Department of Electrical, Computer and Systems Engineering, Rensselaer Polytechnic Institute, Troy, NY e-mail: [email protected] J.H. Tao, T.N. Tan (eds.), Affective Information Processing, 159 c © Springer Science+Business Media LLC 2009